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Abstract 


Warfighter's  mission-critical  system:  Agile  automated  testing  and  test  management 

Do  any  of  these  problems  sound  familiar? 

-  Too  many  failed  scripts 

-  Slipping  schedules 

-  Automation  tools  that  never  get  off  the  shelf 

-  I  ncomplete  test  coverage 

-  Do  more  with  less,  and 

-  Be  creative  in  performing  the  business 

Then,  come  and  listen  to  the  story  of  Warfighter's  mission  critical  application  testing. 

This  case  study  reveals  how  testers  used  automated  functional,  performance,  and 
Service  Test  scripts  for  a  major  mission-critical  application  that  had  to  meet  the 
most  rigorous  quality  standards. 

Also,  working  in  an  agile  environment,  the  team  managed  end-to-end  requirements 
and  defects  and  performed  functional,  SOA  and  performance  testing. 

With  risk  based  and  automated  test  strategies,  the  team  was  able  to  do  automated 
regression  and  smoke  tests  in  a  fast-paced  development  -  agile  environment  and 
managed  test  results,  test  sets,  and  test- related  artifacts.  It  was  a  very  successful 
project,  completed  on  time  with  very  limited  resources. 
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General  I  nfo  for  Testing  &  I  ntegration  in 
Government  Contracting  and  Test  Automation 


Keeping  Track  of: 

Requirements 

Defects 

Test  cases,  test  processes  and  test  plan 

Managing  Testing  Cycles 

Problems  in  infrastructure  and  scalability 

Ongoing  implementations  and  significant  development  initiatives 
Significant  degree  of  customization  and  integration 
Limited  resource  availability 

Integration  with  Complex  Portal  and  Identity  Management  Solutions 

Weak  Testing  Methodology 

Manually  intensive 
People  -  Not  process  driven 
No  automated  testing  capabilities 
Not  trusting  anyone  else  to  test 


Testing  Story 


•  When  a  major  government  contractor  delivers  software,  that  software  must 
comply  with  the  most  rigorous  quality  standards 

•  By  enabling  both  automation  and  proper  test  management,  we  benefit  from 
critical  advantages  not  offered  by  manual  testing 

•  If  in  "agile"  development  model:  Development  cycles  are  short  and  tests 
are  conducted  at  the  same  time  as  the  coding 

•  Things  can  get  particularly  complicated  when  the  team  is  testing  SOA 
applications  and  performance- scalability  testing 
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Why  Testing  is  Very  Crucial  in  Agile  & 
Waterfall  Development? 
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We  Did  Not  Have  Processes  and  Tools  in  Place 


•  The  client  had  Rational  tools  (Req  Pro,  ClearQuest  and  Test  Manager)  but 
they  were  not  properly  used. 

•  And  a  lot  of  Excel  sheets  and  Word  docs.  Never  ending  story  of  not  being 
able  to  control  requirements  and  defects  ... 

•  We  bought  Quality  Center,  and  we  made  Rational  tools  and  Quality  Center 
tools  talk  to  one  another.  We  faced  challenges  with  Firewalls  and  security. 

•  Meanwhile,  we  started  working  on  the  processes  with  the  client  and  our  NG 
internal  processes  for  test. 

•  We  started  putting  together  I  MS,  built  partnership  with  the  client . 
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Agile  Workflow  and  Tools 


Requirements  provided  by 
DISA 


Agile  Sprint  Testing 
Workflow 


NG  Team 
Develops  Use 
Case 


Agile  Team 
Develops  User 
Story 


Tester  Develops  Test 
Cases  Based  on  Input 
from  Agile  Team 
Members 


Fails 

Smoke  Testing 

Fails 

Patch  Testing 

Fails 

Regression  Testing 

hi 

Fails 

CFS  Smoke  Test 

Test  Cases 
Placed  in  Rational 
Test  Manager, 
DISA 


Test  Results  Compiled 
and  Submitted  to  Test 
Manager,  Release 
Manager  and  CM 


Test  Results  Delivered  to  DISA 


*  In  Agile  Development,  test  cases  are  based  on  the  input  from 
Use  Case,  User  Story  and  Agile  Team.  As  they  complete  the 
N  build,  it  it  tested  and  final  test  cases  are  formed 
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Automating  Test  Scripts 


•  Functional  Testing 

•  Performance,  Load,  Stress  Testing 

•  Service  Test 

•  Security  and  SA  Type  of  Testing 

•  I  important  Features  in  Relation  to  Automation: 

-  Test  Case,  Requirement  correlation 

-  Defects,  Defect  Management  and  its  correlation  to  requirements  and  test 
cases  if  possible 

-  Test  Scheduling  I  MS  and  test  schedules  and  customer  testing 
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Purpose  of  Automated  Testing 


•  Checks  virtually  any  functionality  in  application. 

•  Provides  consistently  re-usable  tests  to  be  run  each  time  a  site  or  application 
changes 

•  Shortens  testing  time  especially  regression  testing. 

•  Tracks  all  test  runs,  logs,  test  results  in  one,  shared  repository. 

Major  benefits  are: 

•  Reusability 

•  Consistency 

•  Productivity 

•  Team  work  environment 
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Good  Automation  Candidates 


•  Tests  that  need  to  be  run  for  every  build,  sometimes  referred  to  as  Sanity 
tests  (Smoke  and  Regression  tests). 

•  Tests  that  use  multiple  data  values  for  the  same  actions  are  known  as  Data- 
Driven  tests  (Equals  to,  =>,  <=  ). 

•  Identical  tests  that  need  to  be  executed  using  different  browsers  (We  are 
using  IE6,7  and  FF). 

•  Mission  critical  pages  (Certain  pages  need  to  be  checked  all  the  time). 
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Agile  Testing  Cycles 


•  Sprints  Testing:  For  every  sprint,  test  team  will  have  a  baseline.  The  baseline 
consists  of  tests  created  as  a  result  of  sprint  requirements  that  will  be  fulfilled. 

•  Smoke  Testing:  For  every  sprint,  we  review  the  new  test  cases  and  adjust 
standard  smoke  tests  to  reflect  any  needed  changes. 

•  Regression  Testing:  For  every  sprint,  we  review  the  new  test  cases,  and 
based  on  the  requirements  and  development,  we  complete  a  regression  test. 
That  full  regression  test  includes  all  the  sprint  baseline  regression  and  smoke 
tests.  The  regression  test  is  fully  automated  with  QTP. 

•  Load  Testing:  During  sprints  at  Herndon,  based  on  the  needs,  we  develop 
LoadRunner  scripts  for  performance  and  tuning.  At  the  end  of  each  iteration, 
our  goal  is  to  have  a  set  of  LoadRunner  scripts  that  will  allow  us  to  see  the 
performance,  load  and  scalabity  for  major  business  rules  and  transactions  or 
identify  bottlenecks. . . 

•  Service  Testing:  During  sprints  at  Herndon,  based  on  the  needs,  we  develop 
services  tests  scripts.  Our  goal  is  to  run  these  scripts  under  load  as  well  as 
security. 
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I  n  One  I  teration:  Smoke  and  Regression  Tests 


•  Total  Number  of  Test  Cases 

•  Smoke  -  87  per  browser 

•  Regression  -  259  per  browser 

•  Patch  -  approx  15  per  browser 

•  Total  Number  of  Releases 

•  18  so  far,  with  additional  releases  to  some 
rounds 


Days  and  Resources  to  Test 


■upraDr  uSually  2  peel 

Patch  -  4  hours  per  browser 
(uninterrupted),  usually  1  person 


For  each  release,  we  have  —1000  test  cases  to 
be  executed,  with  5  to  6  resources  over  200 
hours  each. 


This  does  not  include  downtime  or  any  technical 
problems. 


With  the  following  assumptions: 

-  The  testers  are  very  familiar  with  the 
system. 


IllXl  Uwe™o  ae?cr  i  ptions  and  defects 
are  already  in  the  system. 


•  Number  of  cycles  before  ATRR 

•  Herndon 

•  Client  Suite  A 

•  Client  Suite  B 


•  Hours  of  Smoke,  Patch  and  Regression 

•  Herndon  -  4  days  per  release 

•  Client  -  Suite  A  -  3  days  per  release 
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Testing  Documents 


•  Test  Plan 

-  Living  document  updated  throughout  iteration 

-  User  stories  augment  the  test  plan 

-  Delivered  at  the  end  of  each  iteration 


-  Written  throughout  the  Agile  process 

-  I  nput  to  Rational  prior  to  end  of  each  sprint 

-  Automate  QTP,  LR  and  ST 

•  Test  Results 

-  At  Herndon  with  Agile  Teams 

-  Delivered  at  the  end  of  each  sprint 

•  System  Test  Report 

-  Living  document  updated  throughout  iteration 

-  Updated  at  the  end  of  each  sprint 

-  Delivered  at  the  end  of  each  iteration 
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Script  Development 


•  Developed  global  scripts  that  can  be  called  from  one  script  to  the 
other 

•  Scripts  were  grouped  into  test  sets  for  different  purposes.  Such  as 
quick  regression  test  sets,  quick  check  for  critical  areas  or  known 
issues 

•With  one  script  we  were  able  to  test  the  system  with  different 
browsers  at  the  same  time.  E.g.,  I E6,  I E &,  FF3,  etc. 

•  The  same  scripts  were  used  for  executing  tests  at  different  suits.  So 
with  one  script  we  were  able  to  run  several  tests  depending  on  the 
situation  we  were  in  for  that  particular  day. 


Calling  Scripts  from  Other  Test  Sets 
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Calling  scripts  from  other  test  sets,  excluding  log  in  and  log  out, 
preparing  test  sets  and  making  sure  users  have  the  right  privileges  to 
perform  certain  business  rules 


^  HP  Virtual  User  Generator  -  [NG_70I_AV_DODAACLkp_Rprt_Mar5_09  -  Web  (HTTP/HTML)] 

■  _in|x| 

File  Edit  View  Insert  Vuser  Actions  Tools  Window  Help 

1  ’  1  ’’  1  El  i  Jv  C&  *o  I  ("p  -O  ♦!!  i  •  Start  Record  ►  ■  II  £}  |  [jj 

i  1  U)  41  iiP 

\ujl 

l!T  j  Tasks  |  |=I]  Script  |  f|C  Tree  ||S]|  3  iH  03 

f  Start  Page  |  NG 701 AV DODAACLkp Rprt Mar5 09  -  Web  (HTTP/HTML)  | 

<  >  X  1 

web_url ( " logout . j  sp " , 

"URL=https : //{HOST}/gcsspor tal/communi tyF 1 les/shel 1/logout . j sp?cominuni ty=gcssportal%2Fgcss?<2F {PrimaryCSA} " 
"TargetFrame= " , 

"Resource=0 " , 

"RecContentType=text/html “ , 

ue&_pageLabel = gcss_quer y_r epor t s_p 


Edit  with  Notepad...  |  Data  Wizard...  |  Simulate  Parameter... 

Sill 

File  format- 

(•  By  number:  |4  Tj 

Column  delimiter:  | Comma 

Byname:  ^ | 

First  data  Jine:  |l  -jj 

nager/gcss/ { Pr i mar y CSA } ?_n f pb=  true& 
r /gcss/ { Pr i maryCS A } ?_n  f  pb =  t  r ue&_page 
/goss-/  (  Pr  i  maryCS  A  }  ?_n  £  pb  - 1  rue£_pagel 
er /gcss/ { Pr i mar yCSA } ?_n f  pb =  t  rue&_pac 
er /gcss/ { Pr i mar yCSA } ?_n f  pb =  t  rue&_pac 
r /gcss/  {  Pr  ima^CSA^_n  f  pb  =  t  r ue&_page 
ger/gcss/  -flBWIBWHST  ?_n  f  pb = t  r ue&_pe 
ss/{Pr imaryCSA}  ?_n£  pb=  true&_pageLabe 
pmanager/gcss/ { Pr i mar yCSA } ?_n  f  pb  =  t rv 
manager /gcss/ {PrimaryCSA } ?_n f  pb  =  t  r ue 
ppmanager /gcss/ { Pr i mar yCSA } ?_n  f  pb = 1 1 
manager /gcss/ { Pr i mar y CSA } ?_n  f  pb  =  t  rue 
anager/gcss/ {Primary CS A } ?_n  f  pb =  t rued 
pmanager/gcss/{Pr imaryCSA} ?_n£  pb=  trv 
er /gcss/ {PrimaryCSA} ?_n f  pb  =  t  r ue&_pac 
1 /appmanager/gcss/ { Pr i mar yCSA } ?_n  £  pt 
,  ppmanager /gcss/ {Primary CSA }?_nf pb=ti 
nager/gcss/ {PrimaryCSA } ?_n f  pb  =  t  r ue& 
nager/gcss/ {  Pr  i mar yCSA } ?_n f pb=  true& 


Virtual  User  Script  started 
Starting  action  vuser_init  __ 

Web  Turbo  Replay  of  LoadRunner  9 
Run  Mode:  HTML  [Msgld:  MMSG-26^ 

Run-Time  Settings  file:  "D:\PerfTes 


vuser_ini t . c ( 3  3 ) : 
vuser_init . c(43) : 
vuser_ini t .  c  ( 4  6 ) : 
vuser_init . c(46) : 
vuser_init . c(46) : 
vuser_init . c(46) : 
vuser_ini t . c ( 4  6 ) : 
vuser_ini t . c ( 4  6 ) : 
vuser_init . c(46) : 
vuser_init . c(46) : 
vuser_init . c(46)  : 
vuser_ini t .  c  ( 4  6 ) : 
vuser_ini t . c ( 4  6 ) : 


Notify:  Transact 
web_add_au  t  o_head 
Notify:  Parametei 
Warning:  Client  C 
t=355ms:  408-byte 
HTTP/1.1  200 
X-Backside-Ti 
Cache-Control 
Date:  Wed,  IE 
Con tent -Type : 
Consen  t  Requ i i 
Set-Cookie :  d 
24462;  path=/ 


\j£  &  SI  L3  ^  ”  I  ^1  HP  Virtual  User  Gener... 


|  Same  line  as  Username 

“3 

|Each  iteration 

“3 

|  Continue  with  last  value 

~3 

m  Replay  Log  TsT  Recording  Log  |  Correlation  Ri 

Virtual  User  Script  started 
Starting  action  vuser_init. 

Web  Turbo  Replay  of  LoadRunner  9.10.1;. 
Run  Mode:  HTML  [Msgld:  MMSG-2601 

Run-Time  Settings  file:  ''D:\PerfTest 


Select  nr 

Update  value  or 


-Allocate  Vuser  values  in  the  Controller- 
(•  Automatically  allocate  block  size 
C  Allocate  |  values  for  each  Vuser 


vuser_ini t . c ( 3  3 ) 
vuser_ini t . c ( 4  3 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t . c ( 4  6 ) 
vuser_ini t . c ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 


Notify:  Transaction _ 

web_add_au t o_header ("Accept -Language " )  was  success f u 1  [ Msg I d :  MMSG- 26392] 

Notify:  Parameter  Substitution:  parameter  "HOST"  =  "158.114.68.208" 

Warning:  Client  Certificate  was  requested  by  SSL  server 
t=355ms:  408-byte  response  headers  for  "https : //158 . 114  < 

HTTP/ 1  1  200  0K\r\n 
X-Backside-Transport :  OK  0K\r\n 
Cache-Control :  no-cache= "set-cookie "\r\n 
Date:  Wed,  18  Mar  2009  22:14:38  GMTVrVn 
Content-Type:  text/htmlNr\n 
Consen  tRequired:  true\r\n 

Set-Cookie :  def_clus_JSESSIONID=RbSvJByTVBMhpf  t9yBnBZrtChpGLnqMLHrhgy6GFkGsFVvc25cmH 111914 
24462;  path=/\r\n 
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■ao*  »ia  HP  Virtual  User  Gener... 


[Ul“ 


[Dp  Quick  Test  Professional  -  [D:\Program  Files\Mercury  Interactive\QuickTest  Professional 


:  File  Edit  View  Insert  Automation  Resources  Debug  Tools  Window  Help 

i  £jNew  -  Open  -  U  □  I  Yo  ^  I  1  a  ,  1 \fj\  ffll  ffl]  7 
•  ®  Record  ►  Run  ■  Stop  |  I  aii  -  |  u>  j»i|j 


iedule  QTP  Scripts,  Defects 

NORTHROR  GRUMMAi 

3  Mercury  Quality  Center  9.0  -  Microsoft  Internet  Explorer 

File  Edit  View  Favorites  Tools  Help 

i  * 

|  O Back  "  O  iZl  Is]  ^  /  Search  ft  Favorites  ^  ^  □  & 

1  Address  |e]  http://rsmv26-mqc:8080/qcbin/start a.htm 

3  IS Go  Links  >y 

MERCURY 

<  BACK  NEXT  >  TOOLS  ▼  HELP  ▼  |  LOGOUT  | 
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o 

Addre 

Got 


Qu 


Requ 


vjJ  BulkFuellnventory- Jan30 


Item 

Operation 

Value 

▼  0  Actionl 
▼  f§  Browser 

►  f^|  Choose  a  digital  certificate 

►  19  Security  Alert 

►  Q  GCSS  Login 

►  O  GCSS  Login. 

►  ^  GroupS  pace 


Quality  Center 


Domain:  GCSSDOMAIN,  Project:  GCSSTest,  User:  administrator 


Requirements 


Test  Sets  Edit  View  Tests 

Analysis 

4  li  x 

jli  Select  Tests  0  Run  *  Run  Test  Set  X 

| ^  ES-|  © 

Root 

0 Unattached 

$ . _rJ  AppSec 

0  _j  CombinedReports 

0 . EBB 

$ . _J  GA 
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Execution  Grid  Execution  Flow  Test  Set 
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▼  H  Security  Infor 
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▼  GCSS.3 
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Bulk  R 
&  Bulk  R 

▼  g&  inputFram 
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s3 


□  21 


Goggle  |:GV 


-r-|  Go  ■■  ■<  M  ▼  *j(i  ^  Bookmarks  ▼  §15  blocked  ^  Check  ▼  ^AutoLink  ▼  ^  Send  to  ▼ 
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►I  .  Keyword  View 


A1  || 

A 

B 

1 

! 

| 

2 

3 

4 

5 

G 

7 

8 

9 

10 

11 

12 

13 

14 

15 

IB 

17 

18 

19 

20 

21 

Global  A  Actionl 

S8  Data  Table  fllnformatio 


|  GCSS  Demo 
j  Manual  Tests 


LiL 


T 


X  Discussions  "  J  @  ( 


Quality  Center 


<  BACK  NEXT  >  TOOLS  ▼  HELP  ▼  LOGOUT 


Domain:  GCSSDOMAIN,  Project:  GCSSTest,  User:  administrator 


Defects  Edit  View  Favorites  Analysis 


!{*t  New  Defect...  X  O  7  "  i 


m 


jag]  Defect  Details 


Jnl  xl 


Requirements 

4Sli 


I  *4 

Defects 


LA\ 

Dashboard 


|0 

! 

Defect  1C 

Status 

Assignee 

62 

Closed 

mgautam 

mgautam 

64 

Closed 

mgautam 

65 

Closed 

mgautam 

66 

Closed 

pkeith 

0 

67 

Closed 

Inguyen 

0 

68 

Closed 

mgautam 

0 

69 

Closed 

randerso 

0 

70 

Closed 

randerso 

0 

71 

Closed 

randerso 

0 

72 

Closed 

randerso 

73 

Closed 

randerso 

0 

74 

Closed 

randerso 

75 

Closed 

mgautam 

0 

76 

Closed 

randerso 

01 

77 

Closed 

randerso 

0 

78 

Closed 

randerso 

0 

79 

Assigned 

dobbie 

80 

Closed 

Inguyen 

81 

Closed 

mgautam 

0 

84 

Assigned 

dobbie 

I  !  m  *  dP 


Details 

ti 

Description 


A; 

A 

Linked  Entities 

* 

History 


WB  -  Error  when  click  on  delete  Aggregate. 


Details 


*  Build  Reference 
*  Detected  in  Version: 

*  Status: 
*  Test  Case  Reference: 
Associated  Requirement: 

Capability: 
Fix  Due  Date: 
NG.RationalJD: 
Reproducible: 
Use  Case: 


*  Detected  By: 
*  Detected  on  Date: 


*  Subject:  |WB 
Assigned  To:  |mga 
Browser:  | 
Defective  Test  Case:  | 
Modified: 
Priority: 

Suite  Tested: 

User  Story: 


BE 


BE 


8/18/2007  8:53:54  PM 


Planned  Closing  Version:  [_ 
Estimated  Fix  Time: 


Description  |  Attachments  *  History 

-  Execution  Report 

WB  -  Error  when  click  on  delete  A 

Description: 


Closed  in  Version:  | 
Actual  Fix  Time:  [ 
Closing  Date: 


3E 


c 


J3 


Select  "Aggregate  Watchboard  Library."  Clicked  on  "Delete  Aggregate  Watchboard"  link  under  Manage  Contents 
on  the  left,  the  error  shows  like  this  .. 

Error  in  template  wb.showDeleteltem.shtml::  Error:  The  method  "getComtextsO"  was  not  found.. 

Line: "  <ISPECT>if(getSize(getContextsQ)!=1)":  wb_showDeleteltem.shtml:68. 


What  is  the  error  recieved? 


Jake  Jacques  <jjacques>,  6/29/2007:  Now  that  Build  4  is  available  for  testing,  Defect  assigned  to  original  tester 
for  verification  of  fix. 


o/4crtnn7. 


T 


|Defect  53  of  350  [Server  Time:  3:17  PM  3/19/2009 


x  Discussions"  [^J  @  ^  *  &  Discussions  not  available  on  http:// 158. 11 4. 68. 223:81 


cj  Done 

^  ^  f  Start)  [  0]  What  is  Qualit.. 


|  *  Start |  |  0]  What  is  Qualit. . .  |  Screenshotsqt...  |  H  3  Adobe  Rea...  »  Spark 


rrrrrfo  Trusted  sites 

-J  |  _.j  3  Internet  E..!  ▼  ’3j_]  5QA  Agile2.ppt  |  &jj  Presentation!  |  Jj  Inbox  -  Micros. , .  J  ns**  3:31  PM 


© 


©  Trusted  sites 
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Performance  -  Load  -  Stress  Testing 


Questions  a  Performance  Test  Should  Answer: 

-  Does  the  application  respond  quickly  enough  for  the  intended  users? 

-  Will  the  application  handle  the  expected  user  load  and  beyond? 

-  Will  the  application  handle  the  number  of  transactions  required  by  the  business? 

-  Is  the  application  stable  under  expected  and  unexpected  user  loads? 

-  Are  you  sure  that  users  will  have  a  positive  experience  on  go- live  day? 
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Performance  Test  Processes  at  NG  Herndon 


Performance  &  Load  Test  Process  in  the  Application  and  at  Northrop  -  Herndon 


Plan  / 

Design 

v  J 


Define  performance 
test  scenarios 

-  Business  process 

-  Existing  performance 
issue 

-  SLA’s 

-  KPI’s 

-  Baseline 


Application  & 

System  Performance 
Requirements 

-  app  usage  profile 

-  user  profile 

-  system  component 
resource  usage  profile 

-  historic  and  projected 
traffic  data 

-  Max  concurrent  users 

-  Peak  hour  throughput 

-  Typical  user  session 
duration 


System  Architecture 
&  Configuration 

-  Network  topology  diag. 

-  ER  diag. 

-  Data  flow  diag 

-  Server  HW/SW  specs 

Configuration 

-  TCP/IP  connection 
-#  thread  allocated 

-  Memory  allocation 
(JVM  heap  size,  GC 
setting...) 

-  DB  (connection  pool 
size,  SGA  size,  redo  log, 
etc) 

-  FW  setting  &  capability 

-  Load  Balancing  scheme 


Develop  LR 
Scenarios 


-  Define  LR  architecture 

-  Set  up  performance  test 
env. 

-Transaction  definition 

-  Test  data  (user  acct, 
test  parameters,  test 
data...) 

-  Stand  alone  scenario 
for  individual  report  & 
capabilities 

-Combination  scenario 
with  different  reports  and 
capabilities 


Execute 


Scenario  Run-time  Settings 

-  Number  of  VUs 

-  Test  duration  /  Test  iteration 
Define  rendezvous  points 

-Ramp-up  /  Ramped  own  rate 

-  Think  time 

-  Type  of  Browser  emulation 

-  Browser  cache  setting 
Test  log  setting 

-  Test  groups  set  up 


¥ 


Test  Scenario 
Execute 
(Herndon) 


Test  Monitoring 

-  SiteScope 

•  CPU  usage  &  processor 
queue 

•  Memory  usage  (paging/ 
swapping...) 

•  Server  average  load 

-  Hit  per  second 

-  Throughput 

-  Transaction  response  time 

-  Connections  (total  &  new/ 
closed  per  second) 

-  Web  server  (req/sec, 
connection/sec) 

-  App  server  (queue  length, 
queue  time) 

-  DB  server  (lock-wait,  SGA 
size...) 

-  Running  SQL  scripts  ( SQL 
trace,  buffer  cache  hit  ratio,  ,.) 

Network  delay  monitor 


Analyze/ 

Diagnose/ 

Tune 


To 

Suite 


Application 
&  System 
Tuning 


-  Obtain  baseline 

-  Merge  analysis  graph  & 
capture  correlation  patterns 

-  Identify  bottleneck 
transaction 

•  Response  time 

•  Hit  per  second 

•  Transaction  per  second 

-  Identify  bottleneck  system 
component 

-  CPU  usage 

-  Memory  usage 

-  Throughput  trend 

-  Exam  SQL  trace  explain 
plan  -  Identify  slow  queries 

-  Review  LR  and  server 
logs 


-Application  turning 
(optimize  queries,  reduce 
DB  I/O...) 

-  DB  tuning  (index,  optimize 
statistic  analysis,  SGA  size, 
redo  log  buffer  cache  size..) 

-  System  component 
configuration  optimization 
(connection  pool  setting, 
thread  setting...) 

-  System  architecture 
optimization  (Load 
balancing  scheme,  FW  rule 
and  capabilities...) 

-  Network  optimization 
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Monitor 


While  running  performance  test  scripts  one  can  see  the  actual  response  time  and 
monitor  the  servers  involved  in  the  architecture. 


-  Mercury  LoadRunner  Controller  -  RunReportUnitReadiness.Irs  -  [Run] 


Scenario  Monitors  Results  Diagnostics  Too[s  Help 


sqsh  oiti******  m  mm 


Scenario  Groups 

Group  Name 

Down 

o  1 

Pending 

0 

Init 

o  1 

Ready 
°  1 

Run 

o  1 

Rendez 

o  1 

Passed 

1  1 

Failed 

o  1 

Error 

o  1 

Gradual  Exiting 

0 

Exiting 
°  1 

Stopped  . 

°  J' 

Report Unit Re 

1 

I 

_ 

_ 

_ 

_ 

_ 

_ 

_ 

_ 

_ 

Available  Graphs 


S- Transaction  Graphs 

|T rans  Response  Time 


h  Trans/Sec  (Passed) 

T  rans/Sec  (Failed.Stopped) 

L- T otal  T rans/Sec  (Passed) 

]  Web  Resource  Graphs 
Hits  per  Second 
I  Throughput 

!••••  HTTP  Responses  per  Second 
j-  Pages  Downloaded  per  Second 
I--  Retries  per  Second 
;  Connections 
j-  Connections  per  Second 
I- SSL  per  Second 
]  System  Resource  Graphs 
Windows  Resources 
UNIX  Resources 
Server  Resources 
SNMP 

Antara  FlameThrower 
SiteScope 
]•  Network  Graphs 
L  Network  Delay  Time 
]  Firewalls 

j--  Checkpoint  FireWall-1 
]  Web  Server  Resource  Graphs 
Apache 

!••••  iPlanet/Netscape 
iPlanet  (SNMP) 
i-MS  IIS 


SiteScope  -  Last  60  sec 


00:11:00  00:11:10  00:11:20  00:11:30  00:11:40  00:11:50 

Elapsed  Time  (Hour:  Min:  Sec) 


Hits  per  Second  -  whole  scenario 


00:01:00  00:02:00  00:03:00  00:04:00  00:05:00  00:06:00  00:07:00  00:08:00 
Elapsed  Time  (Hour:  Min:  Sec) 


Scale  T  ransaction 


Security  Pages 


vuser_init_T  ransaction 
Login  Page 

Groupspace  Page  •  Click  GCSS 
GCSS  Portal  Home  Page 
Engineering  Page 


Design 
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ip  HP  Virtual  User  Generator  -  [NG_70i_AV_DQDAACLkp_Rprt_Mar5_09  -  Web  (HTTP/HTML)] 


File  Edit  View  Insert  Vuser  Actions  lools  Window  Help 


i  tjJ  '•  I  -  I  H  i  ji  G§>  *n  I  ♦il  ♦!!  :  •  Start  Record 

Start  Page  ]  NG 701 AV DQDAACLkp Rprt Mar5 09  -  Web  (HTTP/HTML)  ] 


II  CsJ  I  I  U>  4l  #  !  3  1#  i  Tasks  |  ID  Script  |  Tree 


li  tarns 


vuser_end ( ) 

lr_s tar t_ transact ion ( " AV_DODAAC_LKP_99_Logout “ ) ; 
web_url ( " logout . jsp" , 

''URI=https : //{ HOST } /gcsspor t a 1 /commun i t yF i les/she 11/1 ogou t . jsp? commun i t  y = gcsspor t  a 1 X 2  FgcssX 2  F {Pr i maryCSA } " 
"Target Frame = " , 

" Resour ce= 0 " , 

"RecContentType=text/html " , 

" Re f  er er = h 1 1 ps : // { HOST} /gcsspor t  a 1 /appmanager/gcss/ {  Pr  i maryCSA }?_nfpb=t  rue&_page!abe 1 = gcss_query_r epor t  s_p 
”Snapshot=t28 . inf " , 

" Mode = HTML" , 

LAST); 

web_url ( " windowCloser . html_2 " , 

" URL=https : //  { HOST } /ga  t eway Log i nApp/w i ndowC loser . html " , 

"Target Frame = " , 

" Resour ce= 0 " , 

"Ref erer= " , 

" Mode = HTML" , 

LAST); 

1 r_end_ transact ion ( " AV_DODAAC_LKF_99_Logout " , LR_AUTO) ; 
return  0 ; 


{J 


[•)  Recording  Log  ^  Correlation  Results  [=)  Recorded  Event  Log  [?|  Generation  Log 


\n 

\t  <script  language® "JavaScript " >\r\n 

\t\t  function  onButtonClick( )  {\r\n 

\t\t\t\t//  Disable  button  to  prevent  multiple  clicks\r\n 
\t\t\t\t document . loginForm . okButton . disabled® 1 ; \r\n 
\r\n 

\t  \t\t//  Try  to  close  mapping  window  if  it  is  open\r\n 

\t  \t\tclose¥indow( ' app_cont roller ' ) ; \n 

\r\n 

\t\t  }\r\n 
\t  < /script  >\r\n 

\t  < /head  >\r\n 
\r\n 

\t<body  onload® "document . loginForm . okButton . focus ( ) " >\r\n 

\t\t\r\n 

\t\t\r\n 

\t\t<span  class® " topBanner" ></span>\r\n 


* 


Start  |  [j)  S3  ”  |  $>}  HP  Virtual  User  Gener.l 


LoadRunner  Controller  Monitors  and  SiteScope 


SiteScope  Monitors: 


HP  Virtual  User  Generator  -  [NG_701_AV_DODAACLkp_Rprt_Mar5_09  -  Web  (HTTP/HTML)] 

:  File  Edit  View  Insert  Vuser  Actions  lools  Window  Help 

i  -  1  &  ’  1  y  i  X  <3  <3  i  •  Start  Record  ►  ■  II  $  |  10  |  1*9  4l  #  !  Q,  H  i 

!  Tasks  |  |H  Script  |  ff3  Tree  ||MllCi 

f  start  Page  I  ING 701 AV DODAACLkp Rprt Mar5 09  -  Web  (HTTP/HTML)  | 

<1  >  X 

lect_EngineeringTab" ) ; 

ager/gcss/ { Pr i maryCS A }?_nfpb=t rue&_pageLabe 1 =  eng i neer i ng_ 1 " 


|  Parameter  Properties  -  [PrimaryCSA] 


mu 


pmanage  Parameter  type:  I  File 


~3 


File  path:  |  D :  \PerfT  est701  \S  criptsVG  lobal_D  ataVU  sername.  dat 
Add  Column...  |  Add  Row...  |  Delete  Column...  |  Delete  Row...  | 


Browse...  | 


Username  I  Password  |  CertUser  |  PrimaryCSA 

1 

tested  OOteslQWert  008  tester  100  GLOBAL 

2 

testerOI teste  QWert  004  tester  01  N0RTHC0M 

tester02teste  QWert _ 005  TestUser02  GLOBAL 

~T~ 

tester03teste  QWert  005  TestUser03  GLOBAL 

5 

tester04teste  QWert  008  TestUser04  DECCD 

1 

6 

tester05teste  QWert  008  TestUser05  DECCD 

1 

tester06teste  QWert  008  TestUser06  DECCD 

- 

Edit  with  Notepad...  |  Data  Wizard...  | 

Simulate  Parameter... 

Select  column 

File  format 

(•  By  number:  |4  Tj 

Column  delimiter:  |  Comma  jJ 

C  Byname:  | 

First  data  Jine:  |l  — j 

|Same  line  as  Username 

3 

(Each  iteration 

|  Continue  with  last  value 

~3 

Replay  Log  ®  Recording  Log  ^  Correlation  Ri 


Virtual  User  Script  started 
Starting  action  vuser_init. 

Web  Turbo  Replay  of  LoadRunner  9.10. 
Run  Mode:  HTML  [Msgld:  MMSG-2601 
Run-Time  Settings  file:  "D:\PerfTesf 


•Allocate  Vuser  values  in  the  Controll 


(*  Automatically  allocate  block  size 
C  Allocate  |  values  for  each  Vuser 


vuser_init . c(33) 
vuser_init . c( 43 ) 
vuser_init . c( 46 ) 
vuser_init . c(46) 
vuser_init . c(46) 
vuser_init . c(46) 
vuser_ini t .  c  ( 4  6 ) 
vuser_ini t .  c  ( 4  6 ) 
vuser_init . c( 46 ) 
vuser_init . c(46) 
vuser_init . c(46) 
vuser_init . c(46) 
vuser_ini t . c ( 4  6 ) 


Notify:  Transaction 


anager/gcss/{ PrimaryCSA} ?_nf  pb=  true£ 
r/gcss/{Pr imaryCSA} ?_nf  pb=  true&_page 
/gcss/ { Pr i mar y CS A } ?_nf  pb=  true&_pagel 
er/gcss/ {Pr i maryCSA >?_nfpb=t rue&_pac 
er/gcss/ { Pr i maryCSA } ?_n  f  pb = t r ue&_pac 
r/gcss/ { Prirr^^CSA}?  n f pb  =  t r ue&_page 
ger/gcss/  ^BWUBHIMbUT  ?_n  f  pb  =  t  rue&_pa 
ss/ {Pri maryCSA } ?_nf  pb=  true&_pageLabe 
pmanager/gcss/{ PrimaryCSA} ?_nf  pb=  trv 
manager /gcss/ { PrimaryCSA} ?_nf  pb=  true 
ppmanager/gcss/{Pr imaryCSA} ?_nf  pb=  tr 
manage r/gcss/ {PrimaryCSA} ?_nf  pb= true 
anager /gcss/ { Pr i maryCSA }?_nfpb=t  r ue& 
pmanager/gcss/ { Pr i maryCSA }?_nf pb=tri 
er/gcss/ { Pr i maryCSA } ?_n f  pb  =  t  r ue&_pac 
1 /appmanager /gcss/ {  Pr  i maryCSA } ?_n  f  pi 
ppmanager /gcss/ {Pr i maryCSA } ?_n  f  pb  =  1 1 
anager /gcss/  {  Pr  i  maryCSA  }?_nfpb=t  rueS. 
anager/gcss/ { Pr i maryCSA } ?_n f  pb =  t rue& 


MMSG-27141 ] 


web_add_auto_header( "Accept -Language " )  was  successful  [Msgld:  MMSG-26392] 

Notify:  Parameter  Substitution:  parameter  "HOST"  =  "158.114.68.208" 

Warning:  Client  Certificate  was  requested  by  SSL  server  [Msgld:  MMSG-26000] 

t=355ms:  408-byte  response  headers  for  "https://158.114.68.208/gcssportal"  (RelFrameId=l,  Internal  ID=1) 
HTTP/ 1  1  200  0K\r\n 
X-Backside-Transport :  OK  0K\r\n 
Cache-Control :  no-cache= " set-cookie " \r\n 
Date:  Wed,  18  Mar  2009  22:14:38  GMT\r\n 
Content-Type :  teHt/html\r\n 
ConsentRequired:  t  r ue\r \n 

Set-Cookie:  def_clus_JSESSIONID=Rb5vJByTVBMhpft9yBnBZrtChpGLnqMLHrhgy6GFkGsFVvc25cmH! 11914 
24462;  path=/\r\n 


*  Start)  ir  Hj  l_3  %  ”  1 1§1  HP  Virtual  User  Gener... 


IjjlSj  3:22  PM 
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Challenges  of  Testing  for  Agile  and  SOA 


A  more  versatile  test-bed  environments 

-  1 1  may  be  difficult  to  model  the  whole  set  of  end-to-end  software  that  probably 
span  many  different  servers 

-  Ability  to  simulate  unavailable  components 

Transition  for  testers  -  process-centric  testing  team 

-  Broad  knowledge  of  business  processes 

-  Understanding  the  intricacies  of  domino  effects  on  business  transa 

-  Cross-functional  teams  environment 

-  Understand  and  diagnose  underlying  technology  and  connectivity 

Location  and  identification  of  web  services  (Geographic  locations...) 


•  Availability  of  web  services  components:  Applications,  Middleware, 
Supporting  hardware,  teams  -  development,  system  admin, 
network,  etc. 


•  Locating  and  isolating  defects  are  difficult: 

•  Defects  in  service  components  would  cause  domino  effects  to 
applications  that  utilize  those  services 

•  Capture  and  analysis  of  all  SOAP  messages  that  are  passed  from  one 
component  to  another  is  overwhelming 

•  Service  components  do  not  have  GUI 
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Testing  Aspects  and  Service  test 


Positive  Testing  -  Generates  a  full  positive  test  for  the  selected  services.  1 1  tests  each 
operation  of  the  selected  service. 

Standard  Compliance  -  Tests  the  service's  compliance  with  industry  standards  such  as  WS-I 
and  SOAP. 


m  Mercury  Service  Test  -  [PositiveTesting  -  Web  Services,  Web  (HTTP/HTML)] 


vith  all 

[ations  using 
ations  using 
;ing  Generic 


site  that  will 

inique. 


Service  I  nteropera 
supported  Wei: 

.NET  Frame 
.NET  Frame 

-  Axis  1.3  We 
Axis  1.3  We 

-  Generic  Mer 
Mercury  Sol 


Security  Testing  - 

-  SQLInjectic 
una 


:  File  Edit  View  Insert  Vuser  Actions  SOA  Tools  Tools  Window  Help 

!  <£jT  ’  I  ”  I  All  :  ®  Start  Record  ■-&  Analyze  Traffic  ►  ;}  '>  i  Tasks  |  .  |  ff3  Tree  I  @1 IEHBH  I:  3  111# 

•  jja  Manage  Services  ^  Add  Service  Call 


Start  Page  [  PositiveTesting  -  Web  Services,  Web  (HTTP/HTML)  |  PositiveTesting  -  Web  Services,  Web  (HTTP/HTML)  | 


0  vuser_end 
»globals.h 


y*  NOTE :  This  is  an  automatically  generated  script,  and  serves  as  a  template. 

■To  optimize  the  use  of  this  script,  it  is  recommended  that  you  review  its  content, 

Irun  it,  examine  its  results,  and  add  checkpoints  as  necessary.  */ 

Act ion ( ) 

L 

Imethod:  getReportList 
■argument  type  string 
■argument  type  string 
■argument  type  string 
■argument  type  string 

|  El  web_service_call (  "StepName=getReportIist_101 " , 

"SOAPMethod=GCSSDetailedReportsService . GCSSDetailedReportsServicePort . getReportList " 
"ResponsePar am = response" , 

" Serv i ce= GCSSDe tail edRepor t  sSer v ice", 

"Snapshot=tl202314686 . inf “ , 

BEG I H_ ARGUMENTS , 

" input =auto  string", 

“GCSSSessionID=auto  string", 

“ GCSSPAR AMETERS = au t o  string", 

END_ARGUMENTS , 

BEGIN_RESULT , 

END_RESULT , 

LAST); 

lr_think_time(3) ; 
return  0 ; 


Extreme  Va 


Null  Values 


For  Help,  press  FI. 

INS  NUM 

f  Start]  J  '^0  jg)  1  S%1  Mercury  Service  Test ... 

|js)  11:18  AM 
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Map  Service  I  nterface  Description 


The  application  has  three  Mapping  I  nterfaces: 

•  Map  Server 

•  Reporting  Detail  Request  from  Palanterra 

•  Reporting  Mapping  Call 


Mapping 
Thrid  Party 

Application 

i 

Map  Server  Request 
Map  Server  Response 


HTTP  Request 


HTTP  Response 
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Reporting  Mapping  Call 


Report  Mapping  Call  is  the  single  internal  Mapping  interface  in  the  application. 

When  a  user  clicks  the  “Map”  button  within  the  results  page  of  a  report,  a  call  is 
made  to  the  Mapping  application. 
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Available  Graphs  x 

E  Runtime  Graphs  * 

Running  Vuseis 
User  Defined  Data  Points 
Error  Statistics 
Vuseis  with  Errors 
E  Transaction  Graphs 

T rans  Response  Time _ 

Tians/Sec  (Passed) 

Trans/Sec  (Failed,S  topped) 

Total  Tians/Sec  (Passed) 

-  Web  Resouice  Giaphs 

Hits  per  Second 
Throughput 

HTTP  Responses  per  Second 
Pages  Downloaded  per  Second 
Retries  per  Second 
Connections 
Connections  pei  Second 
SSL  per  Second 

-  System  Resource  Graphs 

Windows  Resources 
UNIX  Resources 
Server  Resources 
SNMP 

Antaia  FlameThrowei 
SiteScope 
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With  Service  Test  We  are  Able  to: 


•  Develop  scripts: 

-  without  a  GUI 

-  using  multiple  protocols.  I  n  enterprise  world  we  have  to  deal  with  a  lot 
of  multiple  protocols.  This  feature  is  very  helpful. 

-  by  WSDL,  UDDI,  File  and  URL.  This  is  a  very  helpful  feature,  too. 

•  These  scripts  can  be  executed  in  LoadRunner  for  performance 

•  We  can  analyze  traffic  over  the  network 

•  We  can  set  Security  Policies  that  includes  tokens,  SAML,  and  so  on 


28 


Practice  for  Successful  SOA  Testing  Strategy 


Start  early  in  the  life  cycle: 


Testing  client  applications  -  Start  the  end-to-end  testing  and 
tuning  6  months  before  the  deployment. 


■ 


components. 


•  Choose  an  appropriate  set  of  test  cases  to  support  end-to-end 
testing  of  the  business  process  and  end-user  experience. 
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With  the  use  of  the  tools  we  were  able  to: 


•  Prioritize  testing  priorities  based  on  business  risk 

•  Access  testing  assets  anytime,  anywhere  via  a  browser  interface 

•  Create  an  end-to-end  quality  management  infrastructure 

•  Manage  manual  and  automated  tests. 

•  Accelerate  testing  cycles  by  scheduling  and  running  tests  automatically, 
unattended,  24x7 

•  Manage  multiple  versions  of  requirements,  tests,  test  scripts  and  business 
components 

•  Enforce  standardized  processes  and  best  practices 

•  Analyze  application  readiness  at  any  point  in  the  testing  process  with 
integrated  graphs  and  reports 
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With  the  use  of  the  tools  we  were  able  to: 


50  to  70%  decrease  in  actual  testing  time  (efficient  and  faster) 

Able  to  cope  with  huge  amount  of  testing  and  captured  defects  at  early  stages  of 
development 


Able  to  produce  contractual  documents  such  as  RTM  -  Requirement  Traceability 
Matrix,  defect  reports,  test  reports,  test  plans  and  the  like  in  a  timely|  anner. 
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What  We  Need  For  Contractor  I  ntegration  Results 


Purpose:  integrate  and  test  all  system  components  prior  to  official 
delivery  to  the  government. 

S  All  testing  related  documentation  has  been  completed  and  is  up  to  date 
S  Successful  completion  of  smoke,  patch,  and  regression  tests 
S  Performance- Load- Stress  test  baselines  obtained 
S  SLAs  are  met 

S  All  the  test  results  were  delivered  into  government  CM 
S  All  defects  are  documented  in  CM  tool 
S  Final  system  test  report  submitted  to  the  PMO 

S  Installation  and  build  guide  with  all  the  updates  completed  and  delivered 


Results  from  test  indicated  that  the  software  is  ready  for  Acceptance  test 

Requirements  Validation/  Regression  Testing  Results 


Requirements  Validation 

Results  we  would  like  to  see: 

•  Release  Requirements  testing 

-  Completed  99.13%  testing  of  all  testable  requirements 

S  Executed  99.63%  testing  against  IE  6.0 
S  Executed  98.50%  testing  against  IE  7.0 
S  Remaining  0.93%  of  testing  could  not  be  functionally  tested 

•  Regression  testing 

-  Completed  100%  of  planned  regression  testing 

S  Executed  95.05%  testing  against  IE  6.0 
S  Executed  92.95%  testing  against  IE  7.0 


